Forward and backward looking vision system

ABSTRACT

A vision system associated with a projection system includes multiple optical pathways. For instance, when the projection system projects an image onto a generally vertical surface, the vision system may operate in a rear sensing mode, such as for detecting one or more gestures made by a user located behind the projection system. Alternatively, when the projection system projects the image onto a generally horizontal surface the vision system may operate in a front sensing mode for detecting gestures made by a user located in front of the projection system. One or more thresholds may be established for switching between the front sensing mode and the rear sensing mode based on orientation information. As another example, the vision system may be operated in both the front sensing mode and the rear sensing mode contemporaneously.

BACKGROUND

A projection system may project an image onto a projection displayscreen or other passive projection display surface. For instance, theimages may be projected from the front side of a display surface (i.e.,the side facing the viewing audience) or from the rear side (i.e., theside hidden from the viewing audience). With front projection systems,one of the challenges that may impact viewing quality is the physicalarrangement of the screen within an environment, relative to theprojector, and relative to the viewer(s). Ideally, for a conventionalscreen, the projector should project the image from a location that isperpendicular to a planar surface of the screen. The viewer should alsohave a point of view that is normal to the planer surface. However, if aportion of the viewer's body is located between the projector and thedisplay surface, the viewer's body may block at least a portion of theprojected image.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 illustrates an example environmental area, such as a room, inwhich some implementations of the projection techniques and arrangementsdescribed herein may be employed.

FIG. 2 illustrates an implementation of a projection and image capturingsystem including a projector and a camera in spaced relation to oneanother. In this implementation, the projector and the camera havedifferent optical paths.

FIG. 3 illustrates an implementation of a projection and image capturingsystem resembling a familiar type of furniture, such as a table lamp. Inthis implementation, the projector and camera may share a common opticalpath through a common lens.

FIG. 4 illustrates an implementation of a projection and image capturingsystem resembling a table lamp similar to the implementation illustratedin FIG. 3. In this example, the projector and camera may share a commonoptical path through a common lens, and one or more illuminationcomponents may also share the same optical path.

FIG. 5 illustrates a first area of illumination and a second area ofimage projection that may be realized by the examples illustrated inFIGS. 3 and 4.

FIG. 6 shows an exploded view of a head and universal mount of theexamples illustrated in FIGS. 3 and 4.

FIG. 7 illustrates an example projection system with for detectingobjects in a scene according to some implementations.

FIG. 8 illustrates an example projection system operable in a rearsensing mode for sensing gestures according to some implementations.

FIG. 9 illustrates an example projection system operable in a frontsensing mode for sensing gestures according to some implementations.

FIG. 10 is an example flow diagram of a process for controlling a visionsystem according to some implementations.

FIG. 11 is an example flow diagram of a process for controlling a visionsystem according to some implementations.

FIG. 12 is an example flow diagram of a process for controlling a visionsystem according to some implementations.

DETAILED DESCRIPTION

This disclosure describes techniques and arrangements for controlling avision system. Some examples herein include projecting an image onto adisplay surface and using the vision system to detect user gestures,such as for interacting with the image. For instance, a projector, anaugmented reality system, or other type of projection system may projecta visible light image onto a horizontal display surface, onto a verticaldisplay surface, or onto a display surface having some otherorientation. The projection system may also be configured to emitnon-visible light, such as infrared (IR) light, ultraviolet (UV) light,or the like. The non-visible light may be used to illuminate a regionand to reflect off any objects in that region. The reflected non-visiblelight can be captured by the projection system to detect human movement,gestures, and/or expressions. Such gestures may be used in thenavigation and/or operation of the projection system and an associatedcomputing device. For instance, the non-visible light may be projectedfrom the projection system, reflected off of a user's hand, and sensedby a light sensor in the projection system to provide gesturerecognition. As one example, the gesture recognition may enableinteraction with a graphic user interface projected onto the displaysurface.

In some implementations, a user interacts with a projection system thatincludes a combined vision system and projection apparatus. For example,the projection system may project an image onto a horizontal surface,such as a table, or onto a vertical surface, such as a wall or screen.In the table example, the projector projects an image onto the tablesurface, and the vision system may use the same optical path as theprojector for detecting gestures made by the user. Accordingly, thevision system may operate in a front sensing mode when an image isprojected onto a horizontal surface. On the other hand, when theprojector projects an image onto a wall or other vertical surface, thevision system may operate in a rear sensing mode for sensing gesturesmade by a user behind the projector, rather than in front of theprojector. In some cases, the vision system may automatically switchbetween the front sensing mode and the rear sensing mode based onorientation information relating to an orientation of the projector orand/or an orientation of the display surface.

As one example, the vision system may include an infrared (IR) emitter,such as IR LEDs (light emitting diodes) or an IR laser, such as a laserdiode, that project infrared energy to the same general area as theprojected image. The IR light is reflected back to an IR sensor includedwith the projection system. In some examples, the IR sensor may beincluded in a visible light camera, while in other examples, the IRsensor may be a designated IR sensor. A computing device associated withthe projection system may use the reflected IR light to establish adepth map of the field of view of the IR sensor, and identify anydetected gestures using the depth map. Accordingly, the user is able tointeract with the projected image using gestures that are recognized bythe vision system.

In some examples, the projection system may be positioned to project animage toward a vertical surface such as a wall or display screen.However, it is typically not practical for the user to stand next to thewall to interact with the system. For example, the user's head,shoulders, and body may block or otherwise interfere with the projectedimage. Further, some gestures may be blocked by the user's body, and notdetectable by the vision system. Accordingly, implementations hereinenable a rear sensing mode of interaction in which the user may bepositioned behind or adjacent to the projection system for interactingwith the projected image. For instance, the interaction region may be animaginary region in space that is a proxy for the projection surface.Thus, the vision system may include a second vision pathway that emitsnon-visible light in a direction away from the projection direction. Forexample, the second vision pathway may be in a direction opposite to thedirection of the first vision pathway that is optically aligned with thedirection of projection of the image.

As one example, the first and second vision pathways may be mutuallyexclusive such that the front sensing vision pathway is active when theprojection system is projecting onto a horizontal surface, such a table.Furthermore, when the axis of projection is directed toward a verticalsurface, such as a wall, the rear sensing vision pathway may becomeactive and the front sensing vision pathway may be made inactive. Forexample, the projection system may include an accelerometer,potentiometer, or other orientation sensor that detects whether theprojector is directed toward a horizontal surface or a vertical surface.Alternatively, as another example, the projection system may detect adistance to the projection surface or a size of the projected image andmay determine whether to use the front sensing or rear sensing visionsystem based on the determined distance or image size. As still anotherexample, the projection system may automatically detect an orientationof the projection surface and may determine whether to use the frontsensing or rear sensing vision system based on one or more thresholdssuch as an orientation threshold of the projector or the projectiondisplay surface. In still other examples, the first and second visionpathways are not mutually exclusive, but instead may operatecontemporaneously for detecting gestures from both the front and therear of the vision system.

The projection system may include a gesture recognition module that isexecuted on the computing device to allow a user to interact withprojected images, such as graphic interfaces. The vision system field ofview (FoV) may be collinear with the projection direction when theprojector is aimed toward a horizontal surface. Alternatively, when theprojector is aimed toward a vertical surface, the vision system field ofview may be directed in a direction opposite to, at an angle to, orotherwise away from the projection direction. The vision system mayreceive reflected non-visible light to form depth maps that can be usedby the gesture recognition module to recognize gestures, such as may bemade by the hands and fingers of a user. Furthermore, in some examples,the two vision pathways may both use the same light sensor, such as anIR sensor, for forming depth maps. Since the two vision optical pathsmay operate mutually exclusively, they may share the same light sensorfor receiving the reflected non-visible light. Alternatively, of course,each vision system may include its own light sensor, such as an IRdetector, or the like.

In addition, in some implementations, the projection system may includeat least one visible light camera, e.g., an RGB (red, green, blue)camera, that is operable to capture images from the front side whenprojecting onto a horizontal surface and that automatically converts tocapturing images from the rear side when projecting onto a verticalsurface. For example, images from the RGB camera may be used forrecognizing gestures, objects, and user faces, may be used forvideoconferencing, and so forth.

The projection systems described herein may be employed in a variety ofenvironments such as conference rooms, classrooms, homes, offices,commercial environments, retail environments, and so forth. Typicalprojection systems may include a projector configured to emit lightfocused toward a projection display surface. The display surface in turnis configured to reflect and scatter the projected light so that theprojected image is presented to one or more users. The display surfacemay be fixed, such as in the case of a display surface that mounts to awall, table or stand. Alternatively, or additionally, the displaysurface may be portable and freely repositionable, such as a handheldprojection display screen.

In some examples, the projection systems herein may be used in augmentedreality environments that include systems of resources such as cameras,projectors, vision systems, range finders, computing devices withprocessing and memory capabilities, and so forth, which may perform theprocesses described herein. The projectors may project images onto thesurroundings that define the environment or may cause various operationsto be performed within the environment. Moreover, cameras andmicrophones may monitor and capture user interactions with devices andobjects, and these inputs may be used, in part, to determine one or moreimages to present to particular users.

Some implementations may include an augmented reality functional node(ARFN) that is configured to dynamically accommodate motion and tilt inthree-dimensional space. For example, a projector of the ARFN projectslight onto a fixed or mobile projection display surface. In some cases,the display surface may be handheld and may change in one or both of itsdistance from the projector or its angle with respect to an optical axisbetween the projector and the display surface. In response to detectinga change in distance or angle of the display surface, the ARFN maydynamically perform a sequence of actions to accommodate the change.

The systems and techniques described herein may be implemented in manydifferent manners. Several illustrative examples are described below inwhich the projection system is implemented as part of an augmentedreality environment within a room. However, the projection system may beimplemented in many other contexts and situations in which images areprojected onto screens for viewing consumption.

FIG. 1 depicts an example environment 100 in which a projection systemmay be used. The environment 100 may include one or more projectors. Inthe illustrated example, at least one projector is included in an ARFN(augmented reality functional node) 102. However, in other examples, theprojector is not associated with an ARFN, but may instead be astand-alone projector or a projector associated with a different type ofprojection system, display system, media system, computer system, gamingsystem, theater system, videoconferencing system or the like. Forexample, the projectors, vision systems and the display surfaces hereinmay be associated with any type of computing device, home electronics,consumer electronics, automotive electronics, commercial electronics,and so forth.

In FIG. 1, the environment 100 includes three ARFN 102(1)-(3) shownwithin the room. Each ARFN 102 contains projectors, cameras, visionsystems and computing resources that are used to generate the augmentedreality environment 100. In this illustration, the first ARFN 102(1) isa fixed mount system that may be mounted within the room, such asmounted to the ceiling, although other placements are contemplated. Thefirst ARFN 102(1) projects images onto the scene, such as onto a displaysurface 104(1) on a wall of the room. A first user 106 may watch andinteract with the images projected onto the wall, and theceiling-mounted ARFN 102(1) may capture that interaction. In addition,the ARFN 102(1) may detect a location of the user or actions taken bythe user within the room (e.g., gestures) or sounds output by the user.In response, the ARFN 102(1) may identify operations associated withthose locations, gestures or sounds and cause those operations to beperformed within the room. The ARFN 102(1) may further include one ormore devices, such a camera, range finder, or the like, such as todetect a distant to the projected image and/or orientation of thedisplay surface. One implementation of the first ARFN 102(1) is providedbelow in more detail with reference to FIG. 2.

A second ARFN 102(2) may be embodied to resemble a table lamp, which isshown sitting on a horizontal surface of a desk or table 108 in theexample of FIG. 1. The second ARFN 102(2) projects one or more images110 onto a display surface 104(2) of the desk 108, and the user 106 mayview and interact with the projected image 110. The projected image 110may be of any number of things, such as homework, video games, news,movies, television shows, recipes, a graphic interface, and so forth.

A third ARFN 102(3) is also embodied to resemble a table lamp, shownsitting on a small table 112 next to a chair 114. A second user 116 isseated in the chair 114, holding a user device 118. The third ARFN102(3) projects an image onto a display surface 104(3) of the userdevice 118 for the user 116 to consume and interact with the projectedimage. The projected images may be of any number of things, such asbooks, games (e.g., crosswords, Sudoku, etc.), news, magazines, movies,television shows, a browser, a graphic interface, etc. The user device118 may be essentially any device for use within an augmented realityenvironment, and may be provided in several form factors. The userdevice 118 may range from an entirely passive, non-electronic,mechanical surface to a fully functioning, fully processing, electronicdevice with a projection display surface. For instance, the user device118 may be a display surface or display medium that includes one or morefeatures with which the user may interact.

Associated with each ARFN 102(1)-(3), or with a plurality of ARFNs 102,is a computing device 120, which may be located within the augmentedreality environment 100 or disposed at another location external to theenvironment 100. Each ARFN 102 may be connected to the computing device120 via a wired network, a wireless network, or a combination of thetwo. The computing device 120 has a processor 122, an input/outputinterface 124 and a memory 126. The processor 122 may include one ormore processors configured to execute instructions. The instructions maybe stored in memory 126, or in other memory accessible to the processor122, such as storage in cloud-based resources.

The input/output interface 124 may be configured to couple the computingdevice 120 to other components, such as projectors, cameras,microphones, other ARFNs, other computing devices, and so forth. Theinput/output interface 124 may further include a network interface 128that facilitates connection to a remote computing system, such as cloudcomputing resources. The network interface 128 enables access to one ormore network types, including wired and wireless networks. Moregenerally, the coupling between the computing device 120 and anycomponents may be via wired technologies (e.g., wires, fiber opticcable, etc.), wireless technologies (e.g., RF, cellular, satellite,Bluetooth®, etc.), or other connection technologies.

The memory 126 may include computer-readable storage media (“CRSM”). TheCRSM may be any available physical media accessible by a computingdevice to implement the instructions stored thereon. CRSM may include,but is not limited to, random access memory (“RAM”), read-only memory(“ROM”), electrically erasable programmable read-only memory (“EEPROM”),flash memory or other memory technology, compact disk read-only memory(“CD-ROM”), digital versatile disks (“DVD”) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tostore the desired information and which can be accessed by a computingdevice.

Several modules such as instructions, datastores, and so forth may bestored within the memory 126 and configured to execute on a processor,such as the processor 122. An operating system module 130 is configuredto manage hardware and services within and coupled to the computingdevice 120 for the benefit of other modules. In some examples, theoperating system module 130 may present a graphic user interface toenable a user to interact with the ARFN(s) 102 and/or displayed content.

A spatial analysis module 132 is configured to perform severalfunctions, which may include analyzing a scene to generate a topology,recognizing objects in the scene, dimensioning the objects, and creatinga three-dimensional (3D) model or depth map of the scene.Characterization may be facilitated using several technologies includingstructured light, light detection and ranging (LIDAR), opticaltime-of-flight, ultrasonic ranging, stereoscopic imaging, radar, and soforth, either alone or in combination with one another. For convenience,and not by way of limitation, some of the examples in this disclosurerefer to structured light, which may include non-visible (e.g., IR)structured light. Further, in other examples, the light is notstructured light. The spatial analysis module 132 employs theinformation obtained within the augmented reality environment to providean interface between the physicality of the scene and virtual objectsand information. Accordingly, in some examples, the spatial analysismodule may receive an input from the vision system pertaining toreceived non-visible light. The spatial analysis module 132 maydistinguish gestures, control inputs, and the like from otherinformation in the received non-visible light for enabling gesturerecognition.

A system parameters datastore 134 is configured to maintain informationabout the state of the computing device 120, the input/output devices ofthe ARFN 102, and so forth. For example, system parameters may includecurrent pan and tilt settings of the cameras and projectors, anorientation of a projector toward a display surface, distances from theprojector to the display surface, and so forth. As used in thisdisclosure, the datastore includes lists, arrays, databases, and otherdata structures used to provide storage and retrieval of data.

An object parameters datastore 136 in the memory 126 is configured tomaintain information about the state of objects within the scene. Theobject parameters may include the surface contour of the object, overallreflectivity, color, and so forth. This information may be acquired fromthe ARFN 102, other input devices, or via manual input and stored withinthe object parameters datastore 136.

An object datastore 138 is configured to maintain a library ofpre-loaded reference objects. This information may include assumptionsabout the object, dimensions, and so forth. For example, the objectdatastore 138 may include a reference object of a beverage can andinclude the assumptions that beverage cans are either held by a user orsit on a surface, and are not present on walls or ceilings. The spatialanalysis module 132 may use this data maintained in the datastore 138 totest dimensional assumptions when determining the dimensions of objectswithin the scene. In some implementations, the object parameters in theobject parameters datastore 136 may be incorporated into the objectdatastore 138. For example, objects in the scene that are temporallypersistent, such as walls, a particular table, particular users, and soforth may be stored within the object datastore 138. The objectdatastore 138 may be stored on one or more of the memory of the ARFN,storage devices accessible on the local network, or cloud storageaccessible via a wide area network.

In addition, the object datastore 138 may maintain a library of soundsor particular frequencies that are associated with different operationsthat may be performed within the environment. As a result, upon one ofthe ARFNs 102 detecting a particular sound or frequency within theenvironment, the ARFN may identify a corresponding operation (e.g.,adjust volume, project an image to a particular display surface, etc.)and then cause that operation to be performed.

A user identification and authentication module 140 is stored in memory126 and executed on the processor(s) 122 to use one or more techniquesto verify users within the environment 100. In one implementation, theARFN 102 may capture an image of the user's face and the spatialanalysis module 132 reconstructs 3D representations of the user's face.Rather than 3D representations, other biometric profiles may becomputed, such as a face profile that includes key biometric parameterssuch as distance between eyes, location of nose relative to eyes, etc.In such profiles, less data is used than full reconstructed 3D images.The user identification and authentication module 140 can then match thereconstructed images (or other biometric parameters) against a databaseof images (or parameters), which may be stored locally or remotely on astorage system or in the cloud, for purposes of authenticating the user.If a match is detected, the user is permitted to interact with thesystem.

An augmented reality module 142 is configured to generate augmentedreality output in concert with the physical environment. In someexamples, the augmented reality module 142 may employ essentially anysurface, object, or device within the environment 100 to interact withthe users. The augmented reality module 142 may be used to track itemswithin the environment that were previously identified by the spatialanalysis module 132. The augmented reality module 142 includes atracking and control module 144 configured to track one or more itemswithin the scene and accept inputs from or relating to the items. Insome examples, as discussed additionally below, based on input from oneor more sensors, cameras, or the like, the tracking and control module144 may track the position of a user relative to the ARFN 102 fordetermining whether to detect gestures from in front of or from behindthe ARFN 102. For example, when the ARFN 102 determines that the user isin front of the ARFN 102, the ARFN 102 may emit non-visible light towardthe front for detecting user gestures. On the other hand, when the ARFN102 determines that the user is behind the ARFN 102, the ARFN 102 mayemit non-visible light toward the rear for detecting user gestures.

In addition, and as stated above, the memory 126 may maintain, or may beotherwise associated with, a detection module 146. As shown, thedetection module 146 may include an audio detection module 148, an imagedetection module 150 and a gesture recognition module 152. In variousimplementations, a user 106 may interact with the environment for thepurpose of causing one or more operations to be performed within theenvironment. For example, the audio detection module 148 may detect(e.g., via a microphone) sounds or voice commands. Further, the imagedetection module 150 may detect one or more objects, faces, or the like,such as based on input from the spatial analysis module 132 and/or oneor more vision system detectors, cameras or other sensors.

In addition, the gesture recognition module 152 may recognize one ormore actions or gestures performed by the user 106, such as based oninput from the spatial analysis module 132 and/or one or more visionsystem detectors, such as IR sensors, cameras or other sensors. Thegesture recognition module 152 uses various capabilities of the ARFN 102to detect and recognize gestures or other actions made by the user inthe environment 100. The gesture recognition module 152 may process theIR light data and/or perform various types of image processing,including three-dimensional (3D) environment analysis, to detectgestures. The gesture recognition module 152 may further analyzegestures to identify multiple possible candidate gestures, and thendetermine a most statistically probable gesture within the context ofthe gesture, such as based on content currently displayed on a displaysurface with which the user is interacting. Data indicative of detectedgestures may be compared to stored gesture data in datastore 134 toidentify the candidate gestures. When a statistically likely gesture isidentified, the operation associated with the gesture is executed.

Upon detecting a particular action, gesture or other output by a user, aprocessing module 154 may determine one or more operations that areassociated with the detected user output. In particular, the ARFN 102may maintain or be associated with a database that maps various sounds,frequencies, gestures and/or user actions to particular operations thatmay be performed within the environment. That is, in response to theuser 106 performing some action or gesture, the processing module 154may identify a specific operation. If a desired display surface 104 forreceiving a projected image has been identified by the processing module154, a presentation module 156 may cause projection of a particularimage or images onto the display surface. Accordingly, the presentationmodule 156 may cause an image to be projected, which may be caused by auser interacting with the environment.

FIG. 2 shows an illustrative schematic 200 of the first augmentedreality functional node 102(1) and selected components. The first ARFN102(1) is configured to scan at least a portion of a scene 202 and theobjects therein. The ARFN 102(1) may also be configured to provideaugmented reality output, such as images, sounds, and so forth.

A chassis 204 holds the components of the ARFN 102(1). Within thechassis 204 may be disposed a projector 206 that generates and projectsimages into the scene 202. These images may be visible light imagesperceptible to the user, visible light images imperceptible to the user,images with non-visible light, or a combination thereof. This projector206 may be implemented with any number of technologies capable ofgenerating an image and projecting that image onto a surface within theenvironment. Suitable technologies include a digital micromirror device(DMD), liquid crystal on silicon display (LCOS), liquid crystal display(LCD), 3LCD, and so forth. The projector 206 has a projector field ofview 208 that describes a particular solid angle. The projector field ofview 208 may vary according to changes in the configuration of theprojector. For example, the projector field of view 208 may narrow uponapplication of an optical zoom to the projector. In someimplementations, a plurality of projectors 206 may be used. Further, insome implementations, the projector 206 may be further configured toproject patterns, such as non-visible infrared patterns, that can bedetected by camera(s) 210 and used for 3D reconstruction and modeling ofthe environment. The projector 206 may comprise a microlaser projector,a digital light projector (DLP), cathode ray tube (CRT) projector,liquid crystal display (LCD) projector, light emitting diode (LED)projector or the like.

A camera 210 may also be disposed within the chassis 204. The camera 210is configured to image the scene in visible light wavelengths,non-visible light wavelengths, or both. The camera 210 may beimplemented in several ways. In some instances, the camera 210 may beembodied as a red, green, blue (RGB) camera 210. In other instances, thecamera 210 may include time of flight (ToF) sensors. In still otherinstances, the camera 210 may be a red, green, blue, z-depth (RGBZ)camera 210 that includes both IR and RGB sensors. The camera 210 has acamera field of view 212, which describes a particular solid angle. Thecamera field of view 212 may vary according to changes in theconfiguration of the camera 210. For example, an optical zoom of thecamera 210 may narrow the camera field of view 212. In someimplementations, a plurality of cameras 210 may be used, and may face indifferent directions.

The chassis 204 may be mounted with a fixed orientation, or be coupledvia an actuator to a fixture such that the chassis 204 may move.Actuators may include piezoelectric actuators, motors, linear actuators,and other devices configured to displace or move the chassis 204 orcomponents therein such as the projector 206 and/or the camera 210. Forexample, in one implementation, the actuator may comprise a pan motor214, tilt motor 216, and so forth. The pan motor 214 is configured torotate the chassis 204 in a yawing motion. The tilt motor 216 isconfigured to change the pitch of the chassis 204. By panning and/ortilting the chassis 204, different views of the scene may be acquired.The spatial analysis module 132 may use the different views to monitorobjects within the environment.

One or more microphones 218 may be disposed within the chassis 204, orelsewhere within the scene. These microphones 218 may be used to acquireinput from the user, for echolocation, location determination of asound, or to otherwise aid in the characterization of and receipt ofinput from the scene and/or the user device 118. For example, the usermay make a particular noise, such as a tap on a wall or snap of thefingers, which are pre-designated to initiate an augmented realityfunction. The user may alternatively use voice commands for interactionwith the ARFNs. The user may also interact with the user device 118,which may cause the user device 118 to output particular sounds orfrequencies. Such audio inputs may be located within the scene usingtime-of-arrival differences among the microphones and used to summon anactive zone within the augmented reality environment. Further, themicrophones 218 may be used to receive voice input from the user forpurposes of identifying and authenticating the user.

One or more speakers 220 may also be present to provide for audibleoutput. For example, the speakers 220 may be used to provide output froma text-to-speech module, to playback pre-recorded audio, etc.

A transducer 222 may be present within the ARFN 102(1), or elsewherewithin the environment, and configured to detect and/or generateinaudible signals, such as infrasound or ultrasound. The transducer 222may also employ visible or non-visible light to facilitatecommunication. These inaudible signals may be used to provide forsignaling between accessory devices and the ARFN 102(1).

A ranging system 224 may also be provided in the ARFN 102 to providedistance information from the ARFN 102 to an object or set of objects.The ranging system 224 may comprise radar, light detection and ranging(LIDAR), ultrasonic ranging, stereoscopic ranging, one or moreinterferometers, and so forth. In some implementations, the transducer222, the microphones 218, the speaker 220, or a combination thereof maybe configured to use echolocation or echo-ranging to determine distanceand spatial characteristics. Further, any one of, or any combination of,the ranging system 224, the transducer 222, the camera 210, or othercomponents of the ARFN may be used to determine the distance to an imageor a display surface, a size of an image, an orientation of a displaysurface or the like according to the implementations herein.

A wireless power transmitter 226 may also be present in the ARFN 102, orelsewhere within the augmented reality environment. The wireless powertransmitter 226 is configured to transmit electromagnetic fieldssuitable for recovery by a wireless power receiver and conversion intoelectrical power for use by active components in other electronics, suchas non-passive user device 118. The wireless power transmitter 226 mayalso be configured to transmit visible or non-visible light tocommunicate power. The wireless power transmitter 226 may utilizeinductive coupling, resonant coupling, capacitive coupling, and soforth.

In this illustration, the computing device 120 is shown within thechassis 204. However, in other implementations all or a portion of thecomputing device 120 may be disposed in another location and coupled tothe ARFN 102(1). This coupling may occur via wire, fiber optic cable,wirelessly, or a combination thereof. Furthermore, additional resourcesexternal to the ARFN 102(1) may be accessed, such as resources inanother ARFN accessible via a local area network, cloud resourcesaccessible via a wide area network connection, or a combination thereof.

The ARFN 102(1) is characterized in part by the offset between theprojector 206 and the camera 210, as designated by a projector/cameralinear offset “O.” This offset is the linear distance between theprojector 206 and the camera 210. Placement of the projector 206 and thecamera 210 at distance “O” from one another may aid in the recovery ofstructured light data from the scene. The known projector/camera linearoffset “O” may also be used to calculate distances, dimensioning, andotherwise aid in the characterization of objects within the scene 202.In other implementations, the relative angle and size of the projectorfield of view 208 and camera field of view 212 may vary. In addition,the angle of the projector 206 and the camera 210 relative to thechassis 204 may vary or may be variable by actuators.

Due to this offset “O,” the projector 206 and camera 210 employ separateoptical paths. That is, the projector 206 employs a set of lenses toproject images along a first optical path therein, and the camera 210employs a different set of lenses to image the scene by capturing thelight scattered by the surroundings. In other examples, as discussedbelow, the projector 206 and the camera 210 may employ the same opticalpath. Furthermore, the ARFN 102(1) may include one or more IR lightsources 228 for illuminating the scene 202 with structured ornonstructured non-visible light. Accordingly, the vision system may relyon non-visible light in addition to or instead of visible light forperforming functions such as capturing user gestures, recognizing users,detecting objects in the scene, and so forth. In some examples, the IRsources 228 may be a ring of IR LEDs (light emitting diodes) arrangedaround the camera 210 to project IR light toward the scene 202. In otherexamples, the IR source(s) 228 may include an IR laser or any othersuitable source of visible or non-visible light. For instance, thecamera 210 may be capable of detecting IR light in addition to orinstead of visible light.

In other implementations, the components of the ARFN 102(1) may bedistributed in multiple locations within the environment 100. Asmentioned above, microphones 218 and speakers 220 may be distributedthroughout the scene. The projector 206 and the camera 210 may also eachbe located in separate chassis 204.

FIG. 3 illustrates one implementation 300 of the ARFN 102(2) or 102(3),implemented with the appearance of a table lamp, although the componentsmay be incorporated into other types of furniture or other designconfigurations. While not all of the components discussed above areshown in FIG. 3, the ARFN 102 of FIG. 3 may include some or all of thecomponents and functionality discussed above with respect to the ARFN102(1) of FIG. 2. Further, the optical components described in thisimplementation may be embodied in a non-furniture arrangement, such as astandalone unit placed in the room or mounted to the ceiling or walls(i.e., similar to the ARFN 102(1) described above), or incorporated intofixtures such as a ceiling light fixture. The implementation 300 has ahead 302 attached to a base 304 by a movable arm mechanism 306. Asillustrated, the arm mechanism 306 has two base members or rods 308(1)and 308(2) connected to two head members or rods 310(1) and 310(2) via ajoint connector 312. Other configurations of the arm mechanism 306 maybe used.

In the illustrated implementation, the head 302 is connected to the armmechanism 306 via a universal connector 314 that enables at least twodegrees of freedom (e.g., along tilt and pan axes). The universalconnector 314 is described below in more detail with reference to FIG.6. In other implementations, the head 302 may be mounted to the armmechanism 306 in a fixed manner, with no movement relative to the armmechanism 306, or in a manner that enables more or less than two degreesof freedom. In still another implementation, a pan may be coupled to thebase 304 to enable rotation of the arm mechanism 306 and the head 304.

The head 302 holds several components, including a projector 316 and anIR sensor 318. In this example, the IR sensor 318 detects IR lightreflections from objects within a scene or environment. The IR sensor318 may be implemented as a standalone sensor, or as part of a camera210. The head 302 also contains one or more lenses, including a pair offirst lens 320(1)-(2) and a second lens 322. The first lenses 320include a front facing lens 320(1) and a rear facing lens 320(2). Thelenses 320 may be implemented in a number of ways, including as a fixedlens, wide angle lens, or as a zoom lens. When implemented as a zoomlens, the lenses 320 may have any zoom range, with one example being17-50 mm. Use of a zoom lens also offers additional advantages in that azoom lens permits a changeable field of view (FoV), which can increasepixel resolution for better gesture recognition. Further, by zooming in,the device can decrease the field of view and enable the ability todiscern fingers that were not resolved in non-zoomed (larger field ofview) state. The first lenses 320 may further include a motorized focus,a motorized zoom, and a motorized iris (not shown in FIG. 3). The secondlens 322 may be provided to adjust for the differences between theprojection imager (not shown) and the IR sensor 318. This allows for theARFN 102 to set relative coverage of the two imagers (e.g.,overscan/underscan).

The projector 316 projects an image that is reflected off an angled beamsplitter 324 and out through the lens 320. For example, the beamsplitter 324 may be embodied as a dichroic beam splitter having a coatedprism assembly that employs dichroic optical coatings to divide light.For example, the dichroic coating may reflect visible light whileallowing IR light to pass through the coating. Alternatively, in otherexamples (not shown in FIG. 3), the dichroic coating may allow visiblelight to pass through while reflecting IR light. The projected image hasa field of view represented by the outgoing pair of arrows 326. In thismanner, the visible and high intensity light from the projector can bezoomed for image projection on a wide range of surfaces, from near viewto far view surfaces.

One or more IR emitters 328, such as IR LEDs, are positioned in the head302 relative to each of the lenses 320 to emit IR light, as representedby arrows 330. The IR signals are scattered from objects in the sceneand returned to the respective lens 320(1) or 320(2), as represented bythe incoming pair of arrows 332. The captured IR signals are passedthrough the respective lens 320(1) or 320(2) and, on the projection side(i.e., from front facing lens 320(1)), are passed through the dichroicbeam splitter 324 to the secondary lens 322. The IR signals are thenoptionally passed through an IR filter 334 (or other filter type) to theIR sensor 318. In other implementations, the IR signals may be passeddirectly to the IR sensor 318, without going through the IR filters 334.Accordingly, the IR signals are emitted out from the head 302, scatteredby the objects, and collected by the head 302 for capture by the IRsensor 318 as a way to image a scene. The illuminated area may beroughly the same size, or slightly larger, than the area onto whichimages are projected, as is described with reference to FIG. 5 below.This technique may be performed in lieu of using structured light, whichis discussed below with respect to FIG. 4.

A first set 336 of one or more of the IR emitters 328 direct IR light inthe direction of the projected image to illuminate a scene onto whichthe image is being projected. The first set 336 of IR emitters 328 maybe arranged such that the illumination field is wider than theprojection field of view, as represented by the outgoing pair of arrows330, and as further described with respect to FIG. 5 below. Accordinglythe projector 316 shares an optical path with the reflected IR light 332at least through the lens 320(1), i.e., the projected light 326 passesout through the lens 320(1), while the reflected IR light 332 passesinto the projector system through the lens 320(1).

Similarly, a second set 338 of one or more IR emitters 328 direct IRlight in a direction away from the projection direction. Thus, thesecond set 338 of the IR emitters may be arranged to emit IR light in adirection opposed to, or otherwise in a direction away from, thedirection of the projection of the image. For example, when theprojector 316 projects an image onto a generally vertical surface, theARFN 102 may activate the second set 338 of IR emitters to provide rearsensing vision in a rearward direction, or in a direction away from theprojected image.

In this example, a single IR sensor 318 may receive IR signals both fromthe front lens 320(1) and from the rear lens 320(2), since the first setof IR emitters 336 may be operated mutually exclusively of the secondset of IR emitters 338. For example, a suitable mirror arrangement, anoffset arrangement, or the like, (not shown in FIG. 3) may be providedto enable sharing of the IR sensor 318 by the two optical pathscorresponding to the front lens 320(1) and the rear lens 320(2). Inother examples, the IR sensor 318 may be flipped to face whichever ofthe front lens 320(1) or rear lens 320(2) that is expected to receiveemitted IR light. Thus, in some cases, the ARFN 102 may include amechanism for switching the optical components between the front sensingmode and the rear sensing mode.

In still other examples, two IR sensors 318 may be provided, in whichcase, emitted IR light may be sensed from both the front and the rear ofthe projector system simultaneously. Accordingly, the ARFN 102 may beoperated in both the front sensing mode and the rear sensing modecontemporaneously. For instance, multiple users may make gestures bothin front of the ARFN 102 and from the rear of the ARFN 102, such as bytaking turns, at the same time, in synchronized or choreographed motion,and so forth. Further, a single user may move from the rear to the frontof the ARFN 102, or vice versa, and may make gestures at either location(e.g., within a field of view of the front lens or a field of view ofthe rear lens) to interact with the projected image.

As still another example, rather than having two sensors 318 forcontemporaneous front and rear sensing, a single sensor 318 may be usedin an interleaved mode in which images are alternately projected ontothe sensor 318 from the front optical path and from the rear opticalpath. For example, suppose that the sensor 318 is able to capture 120frames per second. Then, the sensor 318 may capture 60 frames or imagesfrom the front optical path, alternating with 60 frames or images fromthe rear optical path. In some cases, operation of the first set 336 ofIR emitters may be alternated in a coordinated fashion with operation ofthe second set 338 of IR emitters in an interleaved strobe-like manner.For example, a signal from the computing device 120 may operate thefirst set 336 to be on for 60 cycles per second and off for 60 cyclesoff per second, while the second set 338 may be operated to be on duringthe time when the first set 336 is off, and off when the first set 336is on. Thus, first IR light from the first set 336 is flashed multipletimes per second in the projection direction, alternating with flashingof second IR light from the second set 338 multiple times per second ina direction opposite to or away from the projection direction. The IRsensor 318 is operated in coordination with the signal driving the sets336 and 338 of IR emitters to receive the reflected first IR light andreflected second IR light, and associate each captured image with eitherthe front optical path or the rear optical path, respectively. Asanother alternative, the sets 336 and 338 of IR emitters may remain inconstant on state, and instead shutters, a spinning mirror, or othersuitable mechanisms (not shown in FIG. 3) may be utilized along theoptical paths for controlling the interleaving of the images receivedalong the front and rear optical paths. Further, while 120 hertz hasbeen provided as an example of frame capture interleaving rate, the ratemay be higher or lower, depending on the characteristics of the sensor318, the IR light emitters, shuttering mechanisms, and the like.

The ARFN 102 of FIG. 3 may further include one or more orientationsensors 340. For example, the orientation sensor 340 may be anaccelerometer that can detect an orientation of the head 302 or a changein the orientation of the head 302 (and thereby a change in orientationof the projector, the axis of projection, and the lenses 320). In otherexamples, the orientation sensor 340 may include a rotational sensor,such as a potentiometer, or the like, located at a pivot point 342 ofthe head 302, and able to detect a change in an angle or orientation ofthe head with respect to the pivot point 342. Numerous other types oforientation sensors 340 will be apparent to those of skill in the arthaving the benefit of the disclosure herein.

Further, in other implementations (not shown in FIG. 3), the projector316 may be arranged to project an image that is passed through the beamsplitter 324 and out through the lens 320, rather than being reflectedby the beam splitter 324. In this arrangement, the returning IR signalsmaybe received back through the lens 320 and reflected by the beamsplitter 324 to the lens 322 and IR sensor 318. In other words, theprojector 316 and IR components (i.e., IR sensor 318, lens 322 andoptionally filter 334) may be swapped so that the returning IR signalsare reflected by the beam splitter 324 rather than the projected image.Accordingly, in this example, the dichroic coating on the beam splitter324 reflects IR light while allowing visible light to pass through. Oneor more additional mirrors may be provided to reflect the IR signalsreceived at the rear side of the ARFN to direct the IR signals to the IRsensor 318, thereby enabling sharing of a single light sensor. Otherarrangements may also be possible where at least part of the opticalpath is shared by the projection and depth capture.

The ARFN 102(2) or 102(3) in the example of FIG. 3 may also be equippedwith one or more components in the base 304. In this example, thecomputing device 120 for executing the spatial analysis module 132 andother modules described above resides in the base 304, along with powercomponents 344 and one or more speakers 220. As discussed above withrespect to FIG. 1, the computing device 120 may include processing andmemory to execute instructions, as discussed above with respect toFIG. 1. The spatial analysis module 132 may be executed by the computingdevice 120 to measure a time of flight for an IR signal (or othermodulated light output). The time-of-flight value may be derived as afunction of a time elapsed between emission from an IR LED 328 andcapture by the IR sensor 318. Alternatively, the time-of-flight valuemay be derived as a function of the phase difference between themodulated light output and the returned light. The spatial analysismodule 132 may be implemented in software or hardware. It is noted thatin other implementations, the components shown as residing in the base304 may reside in the head 302, arm mechanism 306, or elsewhere. Forinstance, the computing device 120 may be located in the head 302, andthe speakers 220 may be distributed in multiple locations, including thebase 304, arm mechanism 306, and/or the head 302. Additionally, in someimplementations, any of the components described above, such as theranging system 224, the transducer 222, or other components may beincluded in the head 302 of the ARFN of FIG. 3.

In the implementation of FIG. 3, the projector 316 and the sensor 318share a common optical path through a common lens 320 on the projectorside. Further, the camera 210 may receive visible light through thefront lens 320(1) along the same path as the projector. Additionally, asecond camera 210 may be provided for receiving visible light along fromthe rear, or alternatively, a second dichroic reflector 324 may beprovided between the rear lens 320(2) and the IR sensor 318 to directvisible light to a single camera 210. Thus, the ARFN 102 of FIG. 3 maybe made more compact to a smaller form factor than that of FIG. 2, asone or more set of lenses may be removed in this design as compared tothe offset design discussed above with respect to FIG. 2. Further, thehead 302 includes a front lens enclosure 346 that contains and protectsthe front lens 320(1), and a rear lens enclosure 348 that contains andprotects the rear lens 320(2). Accordingly, the orientation of thelenses 320 indicates a projection axis and optical axis of the ARFN,which may be used for determining an orientation of the projectionsystem.

FIG. 4 illustrates another implementation 400 of the ARFN 102(2) or102(3), also shown implemented as resembling of a table lamp. Thisimplementation differs from that of FIG. 3 in that the IR illuminationsystem also shares the same optical path as the projector 316 and the IRsensor 318.

In FIG. 4, an IR laser 402 is used in place of the IR LEDs 328 of FIG.3. The IR laser 402 outputs an IR beam that is expanded by a beamexpander 404 and then concentrated by a focus lens 406 onto an angledbeam splitter 408. In one implementation, the angled beam splitter 408is formed of a material that passes light (e.g., glass) and has areflective patch 410 at its center. The focus lens 406 concentrates theIR beam onto the reflective patch 410 of the beam splitter 408, whichdirects the beam through lens 322, through the beam splitter 324, andout through the lens 320. The reflective patch 410 covers the centerportion of the beam splitter 408 and may have any number of shapes, suchas circular, oval, polygonal, and so forth. With this arrangement, thesize and area of interest can be controllably illuminated by use of thelens 320 and modulated IR laser light. The illuminated area may beroughly the same size, or slightly larger, than the area onto whichimages are projected, as is described with reference to FIG. 5 below.

IR signals scattered from a populated landscape are then collected bythe head 302 and passed back through the lens 320, through the beamsplitter 324, through lens 322 (on the projection side), through thenon-reflective portion of the angled reflector 408, through the filter334, and to the IR sensor 318. Accordingly, the collected scattered IRlight may form an image on the IR sensor 318. The image may be used tocompute time of flight values for depth analysis of the landscape of thescene.

When the image is to be projected onto a vertical surface, a mirror 412may be interposed between the lens 406 and the angled reflector 408 todirect the laser beam to a second mirror 414, which reflects the laserbeam to a second angled reflector 408 having a central reflective patch410. The patch 410 reflects the laser beam through the lens 320 on therear side of the ARFN. Thus, in some cases, the ARFN 102 may include amechanism for switching the optical components between the front sensingmode and the rear sensing mode. Further, in the examples in which theARFN 102 is operated in both front and rear sensing modescontemporaneously, a beam splitter or other suitable mechanism ortechnique may be used for directing the laser light to both the frontand the rear contemporaneously. For example, the sensor 318 may becoordinated with the beam splitter for alternately receiving lightreflected from the front and the rear of the ARFN 102 in the interleavedmanner discussed above with respect to FIG. 3. Further, in someexamples, the sensor 318 may include two light sensors such that a firstsensor is positioned to receive reflected IR light through the frontoptical path and a second sensor is positioned to receive reflected IRlight through the rear optical path. Various other techniques andconfigurations that also may be employed for directing the laser throughthe rear lens 320 will be apparent to those of skill in the art in lightof the disclosure herein.

One of the advantages of placing the IR laser 402 as shown and passingthe IR beam through the lens system is that the power used forillumination of a scene may be reduced as compared to the implementationof FIG. 3, where the IR LEDs are external to the optical path.Illumination typically degrades inversely proportional to the square ofthe distance. In FIG. 3, the forward and return paths result in anillumination inversely proportional to the distance to the power offour. Conversely, illumination through the same lens means that thereturned light is inversely proportional to square of the distance, andtherefore can use less intense illumination to achieve the same results.

Further, essentially any IR device may be used in the systems herein.Although IR LEDs and IR lasers are shown in the implementations of FIGS.3 and 4, essentially any device that produces energy within the IRspectrum may be used, such as, for example, a regular red LED.Additionally, in some implementations, any of the components describedabove, such as the ranging system 224, the transducer 222, the separatecamera 210, or other components may be included in the head 302 or otherportions of the ARFN of FIG. 4.

Both implementations of the integrated projection and vision systemafford advantages in addition to a smaller form factor. The projectionand vision system allows for simultaneous and coaxial operation of thefollowing functions: (1) visible light high intensity zoomable imageprojection; (2) illumination of a controlled area of interest withmodulated IR light; and (3) collection of scattered IR light from apopulated landscape to form an image on a time-of-flight camera/IRsensor.

FIG. 5 shows a coverage pattern 500 provided by the ARFN 102(2) or102(3) in the direction of the projection of the image. The coveragepattern 500 has an illumination area 502 covered by the IR-basedillumination system. The coverage pattern 500 also has a projection area504 covered by the projected image. As shown in this footprint, theillumination area 502 is larger than, and encompasses, the projectionarea 504. However, in other implementations, the illumination area 502may be equal to or smaller than, and be encompassed by, the projectionarea 504. The second lens 322 in the device allows for adjustment in therelative IR sensor field of view coverage of the illumination area toenable overscan or underscan conditions. Furthermore, the illuminationarea and field of view of the IR sensor to the rear of the ARFN 102(2)or 102(3) may be the same size, larger, or smaller than that in theforward sensing direction, depending at least in part on the expecteddistance to a user that will interact with the system.

FIG. 6 shows an exploded view 600 of the head 302 and the universalmount 314 of the lamp implementation shown in FIGS. 3 and 4. Here, thehead 302 is generally spherical, although it may be made of any shape,size or form factor. The head 302 has two mounting members 602 onopposing sides of the sphere. The mounting members 602 may be pivotallymounted within a U-shaped cradle 604 to facilitate rotation about a tiltaxis 606. A tilt motor 608 may be included to move the head 302 aboutthe tilt axis 606. In some examples, the tilt motor 608 may be a steppermotor, or the like, that provides an indication of the orientation orangle of the head 302.

The U-shaped cradle 604 is movably mounted relative to structuralbracket 610. The U-shaped cradle 604 may be pivoted about a pan axis612. A pan motor 614 may be included to pivot the U-shaped cradle 604and head 302 about the pan axis 612. Additionally, the U-shaped cradle604 may be rotatable about an axis 616 to rotate or spin relative to thestructural bracket 610. In this example, the head 302 includes the frontlens enclosure 346, which may correspond to the projection side of thehead 302, and the rear lens enclosure 348, which may correspond to anon-projection side of the head 302. Accordingly, the front lensenclosure 346 may be referred to as the forward facing side of the head302 and the rear lens enclosure 348 may be referred to as the rearwardfacing side of the head 302.

FIG. 7 is an illustrative diagram of the ARFN 102 using structured IR toidentify 3D information regarding users, user hands, and other objectswithin an environment. However, while the structured IR light techniquesdescribed herein provide one example for obtaining 3D informationregarding these objects, it is to be appreciated that 3D information maybe determined in other manners in other examples.

In the illustrated example, the projector 316 projects a structured IRpattern 702 onto a scene 704. In some implementations, a sequence ofdifferent structured IR patterns 702 may be used. In otherimplementations, other devices such as general room lighting maygenerate non-visible or visible structured light patterns. A lightfixture, light bulb, or IR source may be configured such that emittedlight contains one or more modulated structured IR patterns 702. Forexample, two structured light patterns may be presented, each at adifferent non-visible wavelength within the structure of an incandescentbulb.

The IR sensor 318 and/or the camera 210 may be used to detect thestructured light, and may also be incorporated into bulbs or assembliessuitable for installation in existing light fixtures. These assembliesmay be configured to communicate with the computing device 120wirelessly or via transmission of a signal via the household electricalwiring. In some implementations, the assembly may provide pre-processingof input prior to sending data along to the computing device 104.

The structured IR pattern 702 may be in IR wavelengths that arenon-visible to the user. In other examples, visible structure light maybe used, or a combination of visible and IR light may be used. Forexample, while the electromagnetic energy used to sense user gestures isdescribed in some examples as IR light, other wavelengths ofelectromagnetic energy may be used, such as visible light, ultravioletlight, or other forms of electromagnetic energy. The structured IRpattern 702 is shown in this example as a grid for ease of illustrationand not as a limitation. In other implementations other patterns, suchas bars, dots, pseudorandom noise, and so forth may be used.Pseudorandom noise (PN) patterns are useful as structured IR patternsbecause a particular point within the PN pattern may be specificallyidentified. A PN function is deterministic in that, given a specific setof variables, a particular output is defined. This deterministicbehavior allows for specific identification and placement of a point orblock of pixels within the PN pattern. In some implementations, aplurality of structured IR patterns 702 may be used to image the scene.These may include different PN patterns, geometric shapes, and so forth.

For illustrative purposes, a sphere 704 is shown positioned between theprojector 106 and a display surface 706 in the scene 202. A shadow 708from the sphere 704 appears on the display surface. Inspection of thesphere 704 shows a deformation or distortion effect 710 of thestructured IR pattern 702 as it interacts with the curved surface of thesphere 704. In some implementations, other effects, such as dispersionof the structured IR pattern 702, may be used to provide information onthe topology of the scene. Where the projector 106 and camera 210/IRsensor 318 have differing fields of view, such as discussed above withrespect to FIGS. 2 and 5, the dispersion or change in the “density” ofthe structured IR pattern 702 may be used to determine depth of field.

The IR sensor 318 and/or the camera 210 may detect the interaction ofthe structured IR pattern 702 with objects within the scene 202. Forexample, the deformation effect 710 on the sphere 704 may be detected bythe camera 210 and the IR sensor 318. Information from the camera 210and/or IR sensor 318 may similarly be used by the computing device 120to identify deformation effects on users within the environment and maythis deformation information may be used to identify user gestures andtrajectories of these gestures. That is, information from the camera 210and/or IR sensor 318 may identify, via deformation in the structured IRpattern 702, a location of a selection tool (e.g., a user's finger orhand) as this location changes over time. The computing device 120 maythen use these locations tracked over time to identify a trajectory ofthe gesture.

FIG. 8 illustrates an example environment 800 in which an ARFN 102, suchas the ARFN 102(2) or 102(3) described above, may be used to observe andidentify hand gestures in a rear sensing mode, such as when projectingan image onto a substantially vertical projection display surface 802.FIG. 8 shows a person's hand 804 as an example of an object within theenvironment that may be analyzed by the ARFN 102. To identify handgestures, the ARFN 102 detects and tracks the hand 804 within a field ofview 806 of the IR sensor 318 and/or camera 210.

As discussed above, one or more modules executable on the computingdevice 120 associated with the ARFN 102 may generate a depth map fromthe vision system information. The depth map may be used to identify thehand 804 of a user, and to determine changes in the location andposition of the hand 804 over time. Specifically, the ARFN 102 mayidentify a sequence of hand positions or poses that form a hand gesturethat is recognizable by the gesture recognition module 152 discussedabove. A hand gesture may be defined by a series of poses of the hand804, where each pose indicates the 3D position of the hand 804 and the3D angular orientation of the hand 804. Position and angular orientationmay be evaluated as absolute positions and orientations or as relativepositions and orientations. As an example, 3D position coordinates maybe specified relative to orthogonal X, Y, and Z axes of a globalcoordinate system for the environment. 3D angular orientations may bespecified as rotations about the X, Y, and Z axes. Furthermore, theorientation of the display surfaces and/or the orientation of the head302 of the ARFN 102 may be determined based on the global coordinatesystem for the environment. For example, a horizontal surface maygenerally be in the plane of the X and Z axes, while a vertical surfacemay generally include the Y axis as a component.

As described above, the IR sensor 318 and/or the camera 210 may be usedin conjunction with a structured IR pattern projected by the IR emittersor other light sources to capture 3D information regarding objectswithin the rear field of view 806. For example, the detected handgestures may enable a user to interact with an image 808, such as agraphic interface, digital content, or the like, projected onto theprojection display surface 802. Thus, the projector 316 may projectlight corresponding to the image 808 in a projection direction onto theprojection display surface 802 within a projector field of view 810.

Furthermore, when in the rear sensing mode, as illustrated, the ARFN 102may emit IR light in a direction away from the projection direction. Forexample, the IR light may be emitted rearward, in a direction oppositeto the projection direction. The IR light that is reflected back fromany objects is detectable within the field of view 806 of the IR sensor318 and/or camera 210. The reflected light pattern can be analyzed toreconstruct 3D characteristics or models of the objects within the fieldof view 806. Accordingly, the rearward sensing mode illustrated in FIG.8 may provide a gesture detection region 812 within the field of view806 of the IR sensor 318. Further, in some cases, the gesture detectionregion 812 may include a virtual wall or selection plane 814 upon whichthe user may make suitable gestures for interacting with the projectedimage 808.

In some examples, the ARFN is able to detect any gestures made by a userat any location within the field of view 806, e.g., without limit to aparticular distance from the ARFN 102, depending on the resolution ofthe IR sensor 318 and/or camera 210. Accordingly, in these examples, theuser may make a gesture anywhere within the field of view 806 forinteracting with the image 808 projected onto the display surface 802.In other examples however, the virtual selection plane 814 may representa virtual plane that the ARFN 102 may reference when determining whetheror not a user is making a selection type gesture. That is, the ARFN 102may define a virtual plane 814 that is substantially parallel to thedisplay surface 802 on which the image 808 is projected and, whenpierced by a gesture of the user, results in a selection being made,such as in a graphic interface included in the projected image.

As one example, the ARFN 102 may define the virtual plane 814 as aselection plane relative to the wall or projection surface 802 on whichthe image 808 is projected. As other examples, the ARFN 102 may definethe selection plane as a certain distance from the ARFN 102 or a certaindistance from the user in the direction of the ARFN 102. In someinstances, the user may define the location of the virtual plane 814 by,for example, providing an audible command or a gesture to the ARFN 102indicating a distance of the selection plane from the user. Of course,in each of these instances it is to appreciated that the virtual wallselection plane might not be visually perceptible to the user, butinstead represents a depth within the environment at which the ARFN 102will interpret a gesture of the user as making a selection or otheraction for interacting with the image 808.

When the user makes a gesture towards the image 808, but prior topiercing to the virtual selection plane 814, the ARFN 102 may providefeedback to the user indicating a portion of the image 808 with whichthe user is currently interacting. For example, if the image includes agraphic user interface, the ARFN 102 may highlight one or more buttonsor items as a user's hand moves from one position to another.Thereafter, if the user's gesture continues moving towards that portionwhen the gesture pierces the virtual selection wall 808, the ARFN 102will interpret the gesture as a selection of that portion of the graphicinterface. Audio clues may be provided contemporaneously to correspondwith movements of the user's hand to further guide the user.

In addition to being used to observe a reflected light pattern in someexamples, as described above, the camera 210 of the ARFN 102 may be usedto capture 2D images of the environment or the user. For example, thecamera 210 may be used in conjunction with ambient lighting to capture a2D image of the user, such as for enabling video conferencing, facialrecognition, and so forth. The captured 2D image may be a color orgrayscale image, comprising an array of pixels defined by tone or colorintensities. Further, some examples may implement 3D shape detection,analysis, and reconstruction using techniques that do not involve theprojection and/or analysis of structured IR or visible light.Accordingly, structured IR or light analysis is described as merely oneexample of various 3D analysis techniques that may be used to identify3D shapes within a scene or within regions of a scene.

In the illustrated example, the projection display surface 802 may be anarea of, or may be located on, a support 816. In some cases, the support816 may be a wall, stand, screen, or any other suitable structure. Inother cases, the support 816 may be a portable support or portabledisplay surface, such as part of a mobile or portable device. The device118 discussed above with respect to FIG. 1 is an example of a portabledevice that may receive and display a projected image in a verticalorientation, a horizontal orientation, or an orientation in betweenvertical and horizontal.

The projection display surface 802 may be any suitable surface capableof receiving and reflecting light projected from the projector 316 todisplay the image 808. In some examples, the display surface 802 may bea display medium such as a reflective sheet of a projection screenmaterial, which may include screens coated with magnesium carbonate,titanium dioxide or other bright reflective material. In other examples,the display surface may be a reflective, lenticular or micro-facetedmaterial, such as acrylic or glass, which provides superior directionaldisplay characteristics. In still other examples, the display surfacemay merely be a surface of a wall or any other suitable surface, and maynot necessarily be a flat surface, but may be a curved surface, apatterned surface, an irregular surface, or the like. For example, thedisplay surface may include at least a portion having a curvature, suchas in the shape of a concave or convex cylinder, hemisphere, etc.Further, the image 808 may be a still image, i.e., a single frame, or amoving image, such as a video that includes multiple frames displayedsequentially.

The ARFN 102 in this example is shown resting on a horizontal surface818, which may correspond to a table, desk, floor, or any other suitablesurface. As mentioned above, the ARFN 102 may include an orientationsensor that automatically detects an orientation of the ARFN or anorientation of a projection display surface 802 upon which the ARFN isprojecting an image. For example, one or more orientation thresholds maybe established at which the ARFN switches from a rear sensing mode asillustrated in FIG. 8, to a front sensing mode. As one example, when theorientation of the display surface 802 exceeds 45° towards vertical, theARFN 102 may operate in the rear sensing mode. Likewise, when theorientation of the display surface is less than 45° the ARFN 102 mayoperate in the front sensing mode. Implementations herein are notlimited to any particular orientation threshold for switching betweenthe rear sensing mode and the front sensing mode, and 45° is only usedas one non-limiting example. As another alternative, the orientation ofthe head 302, i.e., equivalent to the orientation of the projection axisor the optical axis 820, may be tracked for determining when to switchbetween the front sensing mode and the rear sensing mode. As anon-limiting example, when the orientation of the head (e.g.,corresponding to the optical axis of projection 820 and thereby theorientation of the projector) is between horizontal and 45°, the ARFNmay operate in the rear sensing mode. Likewise, when the orientation ofthe head is between 45° and vertical, the ARFN may operate in the frontsensing mode.

Furthermore, in some examples, such as when there are two IR sensors318, the ARFN 102 may operate in both the rear sensing mode and thefront sensing mode contemporaneously, thereby providing one or moreusers with the option to make gestures both in front of and behind theARFN 102. As mentioned above, a single user may move from the rear tothe front of the ARFN 102, or vice versa, and may make gestures ateither location to interact with the projected image. Thus, in someexamples, the rather than switching between a front sensing mode and arear sensing mode, the ARFN may operate contemporaneously in both thefront sensing mode and the rear sensing mode. This also enables multipleusers to make gestures both in front of the ARFN 102 and from the rearof the ARFN 102, such as contemporaneously, by taking turns, and soforth. In this example, the ARFN may be configured to automaticallydetect gestures made from both the front and the rear, and thus, thedetected presence of the user or sensed gestures, rather thanorientation information, may be used to determine a direction or sourceof input gestures. For instance, the tracking and control module 144(discussed above) may determine when a user is within the field of viewof the front or rear optical paths, and the gesture recognition module153 (discussed above) may monitor for gestures input by the userregardless of the orientation of the optical paths. However, in someexamples in which the front and rear sensing mode are operating at thesame time, the orientation information from the orientation sensor(s)may also be taken into consideration when detecting gestures. Forinstance, it may be more likely that a gesture is being made from therear of the ARFN 102 when the optical axis is closer to horizontal thanvertical, and more likely that a gesture is being made from the front ofthe ARFN 102 when the optical axis is closer to vertical thanhorizontal.

FIG. 9 illustrates an example arrangement 900 of the front sensing modeaccording to some implementations. In this example, the ARFN 102 isconfigured to project downward onto a generally horizontal surface 902.The ARFN 102 projects an image 904 onto the display surface 902 within aprojector field of view 906. Furthermore, the ARFN 102 emits IR lightonto the display surface 902 to illuminate an area 908 that isdetectable with in a field of view 910 of the IR detector 318 and/orcamera 210. Accordingly, the head 302 of the ARFN 102 is configured in asubstantially vertical configuration in which the axis of projection 820is substantially vertical for projecting onto the generally horizontalsurface 902. For example, upon detecting that the head 302 has beenpositioned within an orientation threshold, the ARFN 102 mayautomatically begin to operate in a front sensing mode. Thus, the ARFN102 may determine an orientation of the projector 316 and projectionaxis 820, may determine a distance to the projected image 904 or displaysurface 902, may detect a size of the projected image 904, may detect anorientation of the display surface 902, or any combination thereof, fordetermining whether to operate in the rear sensing mode or the frontsensing mode.

In the illustrated example, the IR light is reflected by any objectswithin the field of view 910, such as a user's hand 912. A portion ofthe IR light is reflected back toward the ARFN 102 for detection by theIR sensor 318. Accordingly, in this example, the user may use the hand912 to interact with one or more virtual objects 914, 916 projected ontothe display surface 902 as part of the image 904. For example, the usermay place a finger onto the virtual object 916 for sliding the virtualobject, depressing the virtual object, selecting the virtual object, orthe like. This action by the user is detected by the ARFN 102, and theARFN 102 may modify the image 904 according to the detected gesture andthe current context of the image 904.

As one example, suppose that the user reconfigures the ARFN 102 from theconfiguration of FIG. 9 to the configuration of FIG. 8 (or issues acommand that causes the ARFN 102 to reconfigure itself). Accordingly,upon detecting that the configuration of the ARFN 102 has changed, thepresentation module 156 of the computing device 120 (discussed abovewith respect to FIG. 1), may automatically switch the ARFN 102 from thefront sensing mode to the rear sensing mode as discussed above withrespect to FIG. 8. For example, with respect to the ARFN 102(2), theARFN 102(2) may activate the rear set 338 of IR emitters and deactivatethe front set 336 of IR emitters. Further, if there are any otheradditional conversion steps necessary, such as flipping the IR sensor,moving one or more reflectors into position, or the like, these actionsmay also be performed in response to the detecting of thereconfiguration of the ARFN 102.

Additionally, in some examples, the one or more cameras 210, or othersuitable sensors, may detect the presence of a user in front of orbehind the ARFN 102. In these examples, rather than relying on theorientation information, the ARFN 102 may receive presence informationthat indicates the presence of one or more users in front of or in backof the ARFN 102. For example, the ARFN may initially detect from thepresence information that the user, a user's hand, or other body part isin front of the ARFN, i.e., in the projection direction. Accordingly,the ARFN 102 may emit IR light in the projection direction for detectinggestures made by the user. Subsequently, suppose that the user movesbehind the ARFN 102. Presence information based on information from thecamera(s) 210 or other suitable sensor(s) may indicate that the user isnow behind the ARFN 102, and thus, the ARFN 102 may begin emittingnon-visible light to the rear of the ARFN 102 for detecting gesturesmade by the user from the new location behind the ARFN 102.

Further, in the case that one user is located in front of the ARFN 102and another user is located behind the ARFN 102, the presenceinformation may indicate the presence of both users, and the ARFN 102may emit non-visible light both toward the projection direction and awayfrom the projection direction contemporaneously, as discussed in someexamples herein. Suitable sensors other than the camera(s) 210 fordetermining presence information may include motion sensors, sonar,range finders, LIDAR, depth sensors, and the like. Additionally, the IRsensor 318 may be used for detecting the presence of a user, such as byperiodically flashing IR light to both the front and rear of the ARFN102 for detecting the presence of a user. Other variations will also beapparent to those of skill in the art having the benefit of thedisclosure herein.

FIGS. 10-12 show illustrative processes for controlling a vision systemaccording to some implementations. The processes described herein may beimplemented by the architectures and systems described herein, or byother architectures and systems. These processes are illustrated as acollection of blocks in a logical flow graph. Some of the blocksrepresent operations that can be implemented in hardware, software, or acombination thereof. In the context of software, the blocks representcomputer-executable instructions stored on one or more computer-readablestorage media that, when executed by one or more processors, perform therecited operations. Generally, computer-executable instructions includeroutines, programs, objects, components, data structures, and the likethat perform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described blockscan be combined in any order or in parallel to implement the processes.Further, not all of the blocks are executed in each implementation. Itis understood that the following processes may be implemented with otherarchitectures as well.

FIG. 10 is a flow diagram of a process 1000 according to someimplementations. In some examples, the process 1000 may be executed, atleast in part, by one or more modules of the computing device 120discussed above with respect to FIG. 1.

At 1002, the ARFN may use a projector to project an image in aprojection direction. For example, the projector may project the imagealong a projection axis toward a projection display surface.

At 1004, the ARFN may determine orientation information. For example,the orientation information may be determined based on one or more of anorientation of the projection axis, or the projector, an orientation ofthe display surface, a distance to the display surface, a size of theprojected image, a or a combination thereof.

At 1006, based at least in part on the orientation information, the ARFNemits non-visible light away from the projection direction if anorientation threshold is not exceeded, and emits the non-visible lightin the projection direction if the orientation threshold is exceeded.For example, the vision system may operate in a front sensing mode if anorientation threshold indicates that the orientation of the display iscloser to horizontal, and the vision system may operate in a rearsensing mode if the orientation threshold indicates that the orientationof the display is closer to vertical. Similarly, the vision system mayoperate in the rear sensing mode if an orientation of the projectionaxis or the projector is closer to horizontal, and the vision system mayoperate in a front sensing mode if the orientation of the projectionaxis or the projector is closer to vertical.

At 1008, the ARFN receives at least a portion of reflected non-visiblelight. For example, the reflected non-visible light may pass through therespective front or rear lens to impinge on the light sensor.

At 1010, the ARFN may detect a gesture based at least in part on thereceived portion of the reflected non-visible light. For example, thereceived IR light may be analyzed to determine whether there is anindication of a user gesture and/or to verify an identity of the viewer.In the case of a gesture, human movement may be interpreted as one ormore gestures. Accordingly, the ARFN identifies possible candidategestures, evaluates each gesture to select a most statistically probablycandidate, and then implements the most likely candidate. As anotherexample, in the case of identification verification, the IR light may bedirected at the user's face, with reflected IR light being indicative ofthe facial shape and characteristics that may be used for userauthentication.

At 1012, the ARFN performs an operation in response to the detectedgesture. For example, the operation may be any type of user interactionwith a graphic interface, or display content.

FIG. 11 is a flow diagram of a process 1100 according to someimplementations. In some examples, the process 1100 may be executed, atleast in part, by one or more modules of the computing device 120discussed above with respect to FIG. 1.

At 1102, the ARFN determines at least one of: (1) orientationinformation corresponding to at least one of a projector or a projectiondisplay surface, or (2) presence information indicating a presence of auser within a field of view in a projection direction. For example, theorientation information may indicate an orientation of the projectionaxis and/or an orientation of the display surface such as vertical,horizontal or somewhere in between. Furthermore, the presenceinformation may indicate whether the user is located in front of orbehind the ARFN, such as based on information received from a camera orother sensor.

At 1104, based at least in part on at least one of the orientationinformation or the presence information, the ARFN may emit non-visiblelight in a direction toward a projection direction of the projector. Forexample, suppose that the orientation information indicates that thedisplay surface is closer to horizontal and that the rejection axis iscloser to vertical. Accordingly, the ARFN may operate the vision systemin the front sensing mode by emitting IR light in the same direction asthe projection direction. Similarly, when the ARFN detects that the useris in front of the ARFN, the ARFN may operate the vision system in thefront sensing mode.

At 1106, the ARFN determines at least one of: (1) new orientationinformation corresponding to at least one of the projector or theprojection display surface, or (2) new presence information indicating apresence of the user or another user within a second field of view awayfrom the projection direction. For example, if the ARFN or theprojection surface is moved or reconfigured, new orientation informationmay be determined, such as from a position sensor, camera information orthe like. Similarly, if the user moves to the rear of the ARFN, oranother user is present at the rear of the ARFN, then the ARFN maydetermine new presence information.

At 1108, based at least in part on at least one of the new orientationinformation or the new presence information, the ARFN may emit thenon-visible light in a direction away from the projection direction ofthe projector. For example, suppose that the ARFN or the projectionsurface is reconfigured and the new orientation information indicatesthat the projection surface is now closer to vertical and the projectionaxis is now closer to horizontal. Accordingly, the ARFN may operate thevision system in the rear sensing mode by emitting non-visible light ina direction away from the projection direction. Similarly, if a userdetermined to be present behind the ARFN, the vision system may beoperated in the rear sensing mode.

FIG. 12 is a flow diagram of a process 1200 according to someimplementations. In some examples, the process 1200 may be executed, atleast in part, by one or more modules of the computing device 120discussed above with respect to FIG. 1.

At 1202, the ARFN may use a projector to project an image in aprojection direction. For example, the projector may project the imagealong a projection axis toward a projection display surface.

At 1204, the ARFN may emit first non-visible light in a first directiontoward the projection direction and emit second non-visible light in asecond direction away from the projection direction. For example, thefirst non-visible light may be emitted toward the front of the ARFN,i.e., in a direction toward the projected image, while the secondnon-visible light may be emitted toward the rear of the ARFN, i.e., in adirection away from the projection direction.

At 1206, the ARFN receives, via at least one light sensor, a reflectedportion of the first non-visible light through a first lens and areflected portion of the second non-visible light through a second lens.For example, the reflected portion of the first non-visible light may bereceived along a first optical path including a first lens, while thereflected portion of the second non-visible light may be received alonga second, different optical path including a second lens.

Although the subject matter has been described in language specific tostructural features, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features described. Rather, the specific features are disclosedas illustrative forms of implementing the claims.

What is claimed is:
 1. A system comprising: a projector to project lightcorresponding to an image in a projection direction toward a displaysurface; one or more processors; one or more computer-readable media; atleast one light emitter configured to emit non-visible light; a lightsensor to detect the non-visible light; and one or more modulesmaintained on the one or more computer-readable media which, whenexecuted by the one or more processors, cause the one or more processorsto perform operations comprising: ascertaining whether the projector is(1) oriented so that the projection direction is within a first range ofdirections that includes a substantially horizontal direction or (2)oriented so that the projection direction is within a second range ofdirections that includes a substantially vertical direction; based atleast partly on a first determination that the projector is oriented sothat the projection direction is within the first range of directions,emitting first non-visible light in a first direction away from theprojection direction, receiving at least a first portion of the firstnon-visible light reflected from one or more first objects in anenvironment, and detecting at least one first gesture in a first areaoutside of the first range of directions relative to the projector basedon first information generated from the first non-visible lightreflected from the one or more first objects in the environment; basedat least partly on a second determination that the projector is orientedso that the projection direction is within the second range ofdirections, emitting second non-visible light in a second directiontoward the projection direction, receiving at least a second portion ofthe second non-visible light reflected from one or more second objectsin the environment, and detecting at least one second gesture in asecond area within the second range of directions relative to theprojector based on second information generated from the secondnon-visible light reflected from the one or more second objects in theenvironment.
 2. The system as recited in claim 1, wherein the at leastone light emitter comprises: a first light emitter positioned to emitthe second non-visible light in the second direction toward theprojection direction; and a second light emitter positioned to emit thefirst non-visible light in the first direction away from the projectiondirection.
 3. The system as recited in claim 1, wherein the operationsfurther comprise performing an operation based at least in part on atleast one of the at least one first gesture or the at least one secondgesture.
 4. The system as recited in claim 1, wherein the projectorshares an optical path with the second non-visible light reflected fromthe one or more second objects in the environment at a time at which thesecond non-visible light is emitted in the second direction toward theprojection direction.
 5. A method comprising: using a projector toproject an image in a projection direction onto a surface; determiningthat the projection direction relative to the surface is in asubstantially vertical direction; based at least partly on determiningthat the projection direction relative to the surface is in thesubstantially vertical direction, emitting first non-visible light intoa first gesture area toward the projection direction; receiving a firstreflected portion of the first non-visible light; based at least partlyon the first reflected portion of the first non-visible light, detectingone or more first gestures made in the first gesture area; determiningthat the projection direction relative to the surface is in asubstantially horizontal direction; based at least partly on determiningthat the projection direction relative to the surface is in thesubstantially horizontal direction, emitting second non-visible lightinto a second gesture area away from the projection direction; receivinga second reflected portion of the second non-visible light; and based atleast partly on the second reflected portion of the second non-visiblelight, detecting one or more second gestures made in the second gesturearea.
 6. The method as recited in claim 5, wherein the substantiallyhorizontal direction is on an opposite side of the projector from thesurface in a direction away from the projection direction.
 7. The methodas recited in claim 5, further comprising performing an operation basedat least partly on at least one of the one or more first gestures or theone or more second gestures.
 8. The method as recited in claim 5,wherein determining the projection direction is based at least in parton information from a sensor associated with the projector, wherein thesensor comprises at least one of: an accelerometer; or a potentiometer.9. The method as recited in claim 5, wherein: the projector projects theimage through at least one lens; and a light sensor receives the firstreflected portion of the first non-visible light through the at leastone lens at a time at which the first non-visible light is emitted inthe projection direction.
 10. The method as recited in claim 5, wherein:detecting the one or more first gestures made in the first gesture areacomprises receiving the first reflected portion of the first non-visiblelight that is reflected from at least a first portion of a body of auser; and detecting the one or more second gestures made in the secondgesture area comprises receiving the second reflected portion of thesecond non-visible light that is reflected from at least a secondportion of the body of the user.
 11. The method as recited in claim 10,further comprising emitting the first non-visible light into the firstgesture area and emitting the second non-visible light into the secondgesture area concurrently.
 12. The method as recited in claim 5, furthercomprising: determining a change in the projection direction relative tothe surface from the substantially vertical direction to thesubstantially horizontal direction; and upon determining the change inthe projection direction, detecting the one or more second gestures madein second gesture area.
 13. The method as recited in claim 5, furthercomprising operating a camera to receive visible light from a directionthat is determined based at least in part on the projection directionrelative to the surface.
 14. A system comprising: a first lens to passfirst projected content to and to receive first reflected non-visiblelight from a first field of view in a substantially vertical direction;a second lens to pass second projected content to and to receive secondreflected non-visible light from a second field of view in asubstantially horizontal direction; at least one light emitter toproject light in a projection direction relative to a surface, the atleast one light emitter to: emit first non-visible light in thesubstantially vertical direction into the first field of view, thesubstantially vertical direction being the projection direction; andemit second non-visible light in the substantially horizontal directioninto the second field of view, the substantially horizontal directionbeing away from the projection direction; at least one light sensor toreceive the first reflected non-visible light through the first lens inthe substantially vertical direction and to receive the second reflectednon-visible light through the second lens in the substantiallyhorizontal direction; and a processor configured to receive informationrelated to at least one of one or more first gestures detected based atleast partly on the first reflected non-visible light or one or moresecond gestures detected based at least partly on the second reflectednon-visible light.
 15. The system as recited in claim 14, wherein the atleast one light sensor comprises a single light sensor to obtain firstimages from the first reflected non-visible light received through thefirst lens, wherein the first images are interleaved with second imagesobtained from the second reflected non-visible light received throughthe second lens.
 16. The system as recited in claim 14, wherein theprocessor is further configured to alternately flash the firstnon-visible light in the substantially vertical direction and the secondnon-visible light in the substantially horizontal direction.
 17. Thesystem as recited in claim 14, further comprising a projector to projectthe light corresponding to an image through at least one of the firstlens or the second lens.
 18. The system as recited in claim 14, whereinthe projector is further configured to: determining a change in theprojection direction relative to the surface from the substantiallyvertical direction to the substantially horizontal direction; and upondetermining the change in the projection direction, detecting the one ormore second gestures.
 19. One or more non-transitory computer-readablemedia comprising instructions which, when executed by one or moreprocessors, cause the one or more processors to perform operationscomprising: determining at least one of: first orientation informationcorresponding to at least one of a projector, a projection displaysurface, or the projector relative to the projection display surface; orfirst presence information indicating a presence of at least a portionof a user within a field of view in a projection direction; based atleast in part on at least one of the first orientation information orthe first presence information, emitting non-visible light in a firstdirection toward the projection direction of the projector; determiningat least one of: second orientation information corresponding to atleast one of the projector, the projection display surface, or theprojector relative to the projection display surface; or second presenceinformation indicating a presence of at least a portion of the user oranother user within a second field of view away from the projectiondirection and based at least in part on at least one of the secondorientation information or the second presence information, emitting thenon-visible light in a second direction away from the projectiondirection of the projector.
 20. The one or more non-transitorycomputer-readable media as recited in claim 19, wherein the determiningthe first orientation information is based on at least one of: a firstorientation of an axis of projection of the projector; a secondorientation of the projection display surface that receives a projectedimage from the projector; a size of the projected image; or a distancefrom the projector to the projection display surface.
 21. The one ormore non-transitory computer-readable media as recited in claim 19, theoperations further comprising: receiving, by a light sensor, a reflectedportion of the non-visible light; determining, at least in part from thereflected portion of the non-visible light, a gesture made by the useror the other user; and performing an operation based at least in part onthe gesture.
 22. The one or more non-transitory computer-readable mediaas recited in claim 19, the operations further comprising operating atleast one camera for determining the first presence information and thesecond presence information.
 23. A method comprising: using a projectorto project an image in a projection direction; determining a firstorientation of a projection surface relative to the projector; emittingfirst non-visible light in a first direction toward the projectiondirection if based at least partly on a determination that anorientation of the projection surface is substantially in the firstorientation; receiving, by at least one light sensor, a first reflectedportion of the first non-visible light through a first lens; determininga change from the first orientation of the projection surface relativeto the projector to a second orientation of the projection surfacerelative to the projector; emitting second non-visible light in a seconddirection away from the projection direction based at least in part onthe change; and receiving, by the at least one light sensor, the firstreflected portion of the first non-visible light through the first lensand a second reflected portion of the second non-visible light through asecond lens.
 24. The method as recited in claim 23, further comprisingdetecting a gesture based at least in part on at least one of the firstreflected portion of the first non-visible light or the second reflectedportion of the second non-visible light.
 25. The method as recited inclaim 23, wherein using the projector to project the image furthercomprises using the projector to project the image through the firstlens through which the first reflected portion of the first non-visiblelight is received.
 26. The method as recited in claim 23, whereindetermining the change from the first orientation to the secondorientation comprises at least one of: determining a first change in adistance from the projector to the projection surface; or determining asecond change in an angle between the projector and the projectionsurface.